Backing up and restoring server clusters

Before you begin to back up any data on the server cluster nodes, make sure you backup the cluster disk signatures and partitions using Automated System Recovery in the Backup Wizard. This step is necessary if you later need to restore the signature of the quorum disk, for example, if you experience a complete system failure, and the signature of the quorum disk has changed since you last backed up.

Note

• By default, Backup Operators do not have the user rights necessary to create an Automated System Recovery (ASR) backup on a cluster node. However, Backup Operators can perform this procedure if that group is added to the security descriptor for the Cluster service. You can do that using Cluster Administrator or cluster.exe. For more information, see To give a user permissions to administer a cluster and Cluster.

For information, see To back up cluster disk signatures and partition layouts.

When you back up data on a server cluster node, make sure you also back up the cluster quorum. The cluster quorum is important because it contains the current cluster configuration, application registry checkpoints, and the cluster recovery log.

You can use the Backup Wizard to back up the cluster quorum data if you perform a System State backup from any node provided the Cluster service is running on that node.

For information, see To back up the cluster quorum.

To back up all cluster disks owned by a node, perform a full backup from that node.

You can also back up this data through a network connection to a hidden administrative file share. For example, you might use the New Resource Wizard to create FBackup$, GBackup$, and HBackup$ file shares for the root of drives F, G, and H, respectively. These shares would not appear in the browse list and could be configured to allow access only to members of the Backup Operators group.

For information on backing up data on the cluster disks, see To back up data on cluster nodes.

Important

• If a cluster disk owned by the node being backed up fails over to another node during the backup process, the backup set will not contain a full backup of that disk.
• You can only backup a cluster disk on a local node. You cannot backup a cluster disk on a remote computer.

After you back up the cluster quorum disk on one node, it is not necessary to back up the quorum on the remaining cluster nodes. However, you may want to back up the clustering software, cluster administrative software, system state, and application data on the remaining nodes.

Important

• If you back up the system state for a node, you will also automatically back up the quorum data as long as the Cluster service is running on that node.

For information on backing up data on individual cluster nodes, see To back up data on cluster nodes.

If you have lost files and folders on one of your cluster disks, but not on the disk containing the cluster quorum, you can use the Backup or Restore Wizard to restore that data.

Important

• You must restore the cluster disk data from the node that owns the cluster disk.

For information, see To restore files from a file or a tape.

Symptom: The cluster nodes can boot up, but the Cluster service fails to start because the quorum resource cannot come online.

If this problem results from corrupted files on the quorum disk, follow the steps outlined in To recover from a corrupted quorum log or quorum disk. If the cluster quorum disk needs to be replaced, see Scenario 5, below. For a majority node set cluster, see Scenario 9, below.

Symptom: Some resources fail to come online and the configuration data is out of date.

If you have recovered from quorum corruption by creating a new quorum log as described in Scenario 2 above, you may need to restore the matching checkpoints before the quorum resource can come back online.

Using the Microsoft Windows Server 2003 Resource Kit tools

• Use the ClusterRecovery utility. For more information, see the Microsoft Windows Server 2003 Resource Kit.

Using Windows Server 2003 family tools

• Locate backup sets of the cluster quorum disk and system disk. The cluster quorum disk backup will contain other system state data. These backups need to be a matched set, that is make sure they were taken at the same time.
• Restore data on the system disk. For information, see To restore files from a file or a tape.
• Restore the cluster quorum. For information, see Restore the contents of a cluster quorum disk for all nodes in a cluster.

Symptom: The cluster disk cannot come online. Resources that depend on that cluster disk may also not be able to come online.

First, run the chkntfs command to determine if the disk is merely corrupted. For information, see Chkntfs command.

If the cluster disk is corrupted or the disk hardware fails, you can restore the disk while still keeping your cluster up and running by using utilities in the Microsoft Windows Server 2003 Resource Kit. If you do not have access to these tools, you can still restore your cluster disk using the Backup and Recovery utilities included with Windows Server 2003 family operating systems, but your cluster will not be in operation during the recovery process.

Using the Microsoft Windows Server 2003 Resource Kit tools

• Locate the data backup set for the node that owns the cluster disk. Also, locate the Automated System Recovery backup set for that node, if it is available.
• If necessary, replace the cluster disk. For information, see To install local storage buses and devices.
• If you do have an Automated System Recovery backup set for the node, use the Confdisk utility from the Windows Resource Kit, along with NTBackup to restore the data on the cluster disk. For more information, see the Microsoft Windows Server 2003 Resource Kit.

  Notes

  • If you do not have the Automated System Recovery backup set, you can still restore the cluster disk by using the ClusterRecovery utility from the Microsoft Windows Server 2003 Resource Kit and the /fixquorum Cluster service start up parameter. For more information, see the Microsoft Windows Server 2003 Resource Kit.
  • If you use the Confdisk utility to restore your cluster disks, the checkpoints might not match.

Using the Microsoft Windows Server 2003 Resource Kit tools

• If necessary, replace the cluster disk. For information, see To install local storage buses and devices.
• Stop the Cluster service on all nodes of the cluster.
• Perform an Automated System Recovery restore on the node that owns the failed cluster disk. For more information, see To restore a damaged cluster node using Automated System Recovery.
• After the restored node comes back online, restart the Cluster service on the remaining nodes.

Symptom: The cluster nodes can boot up, but the Cluster service fails to start because the quorum resource cannot come online. Entries in the Event Log indicate hardware failures.

If the cluster quorum disk (the disk containing the quorum resource) fails, you can replace it while still keeping your cluster up and running by using utilities in the Microsoft Windows Server 2003 Resource Kit. If you do not have access to these tools, you can still replace your cluster quorum disk using the backup and restore utilities shipped with Windows Server 2003 family operating systems, but you must first bring down your cluster.

Using the Microsoft Windows Server 2003 Resource Kit tools

• Locate the data backup set for the cluster quorum. Also, locate the Automated System Recovery backup set for the node you used to create the cluster quorum backup set, if it is available.
• If necessary, replace the cluster quorum disk. For information, see To install local storage buses and devices.
• If you do have an Automated System Recovery backup set for the node, use the Confdisk utility from the Microsoft Windows Server 2003 Resource Kit, along with NTBackup, to restore the data on the cluster disk. For more information, see the Microsoft Windows Server 2003 Resource Kit.

  Notes

  • If you do not have the Automated System Recovery backup set, you can still restore the cluster quorum disk by using the ClusterRecovery utility from the Microsoft Windows Server 2003 Resource Kit. For more information, see the Microsoft Windows Server 2003 Resource Kit.
  • If you use the Confdisk utility to restore your cluster quorum disk, the checkpoints might not match.

Using Windows Server 2003 family tools

• Perform an Automated System Recovery restore on any node in the cluster. For more information, see To restore a damaged cluster node using Automated System Recovery.
• Restore data on the node. For information, see To restore files from a file or a tape.

Note

• Alternatively, you can also create a new quorum on another node, restart the cluster using the /fixquorum Cluster service start up parameter, and switch to the new quorum resource. For more information, see Fixquorum command.

Symptom: The node cannot join the cluster.

If the Event Log indicates that the cluster database on the local node is merely corrupted, you can perform a System State restore on that node to replace the local cluster database. For information, see To restore the cluster database on a local node. Alternatively, you can copy the latest checkpoint file (CHKxxx.TMP) from the quorum disk to the %systemroot%\Cluster\ directory, rename it as file CLUSDB, and restart the Cluster service on that node.

If a single node fails in the cluster due to system disk or other hardware failure, follow these steps to rebuild the node and rejoin the cluster:

• After verifying that all cluster resource groups have been successfully moved to other nodes, repair or replace the failed hardware. For information, see To move a group to another node and To manage cluster hardware.
• Perform an Automated System Recovery restore on the failed node to rebuild the node. For information, see To restore a damaged cluster node using Automated System Recovery.
• If you have other files or application data for that node backed up on on tape or other backup medium, you can restore that now. For information, see To restore files from a file or a tape and Scenario 8 below.
• For each cluster group and resource, verify that the newly recovered node appears as a possible owner in Cluster Administrator, then move a resource group to the newly recovered node and verify that the move is successful. For information, see To test whether group resources can fail over.

Note

• If you do not have an Automated System Recovery backup of the node, you can evict that node and add a new node to the cluster. For more information, see To evict a node from the cluster and To add additional nodes to the cluster.

If your cluster is not functioning as expected, you can use the Backup or Restore Wizard to rollback your cluster to a previous configuration.

For information, see Restore the contents of a cluster quorum disk for all nodes in a cluster.

Symptom: None of the nodes can boot up.

If all nodes fail in a cluster and the quorum disk cannot be repaired, follow these steps:

• Use Automated System Recovery on each node in the original cluster to restore the disk signatures and the partition layout of the cluster disks (quorum and nonquorum). For more information, see To restore a damaged cluster node using Automated System Recovery.
• Recreate the original cluster, join all nodes to the cluster, and recreate the original Cluster group. For information, see Checklist: Planning and creating a server cluster.
• Using the Backup or Restore Wizard, restore the cluster quorum on all nodes. For information, see Restore the contents of a cluster quorum disk for all nodes in a cluster.
• Install cluster applications on each cluster node and recreate the cluster resources using the same names as before.
• Restore your application data from backup data sets.

Important

• If you do not have an Automated System Recovery backup of each node, you cannot restore the cluster. Instead, you must recreate your cluster from scratch. For more information, see Checklist: Planning and creating a server cluster.

On a majority node set cluster, the cluster database is not stored on a cluster disk central to all nodes, but is instead stored locally on each node at %systemroot%\Cluster\MNS.%ResourceGUID%$\%ResourceGUID%$\MSCS\. Because the majority node set cluster model does not use cluster disks, you can simply use the Backup or Restore Wizard to restore the data on individual nodes.

Using the full backup set for each node, restore data to each node in the cluster. For information, see To restore files from a file or a tape. The Cluster service will replicate the latest version of the cluster database to all other nodes.

If you want to restore an older version of the cluster database, stop the Cluster service on all nodes of the cluster and delete the local copies of the database (all the files under the \MSCS\ folder) on those nodes. Then restore the cluster database to one node and restart the Cluster service on all nodes. The Cluster service will replicate the restored version of the cluster database to all other nodes.

Note

• If you do not have a backup of the cluster database and/or the cluster database files are corrupted, use the /resetquorumlog Cluster service start up parameter on one node of the cluster to create new cluster database files on that node. Then restart the remaining nodes. For more information, see Resetquorumlog command.

In a majority node set cluster, if some of the nodes fail, and the cluster loses quorum, you can force the remaining nodes to form a quorum and restart the cluster. For more information, see To force quorum in a majority node set server cluster.

When restoring application data in a server cluster, follow the instructions provided in the documentation that shipped with your application.

Important

• If you are backing up Microsoft Exchange Server, a newer version of NTBackup.exe may be available from the Exchange Server section of the Microsoft Web site(http://www.microsoft.com/). Otherwise, you can use the version of NTBackup.exe that is included with Windows Server 2003 family operating systems.